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ABSTRACT 

Urban  Resolve  2005  (UR05)  was  a  human-in-the-loop 
experiment  supported  by  Joint  Semi-automated  Forces 
(JSAF)  simulation  to  replicate  operations  in  an  urban 
setting  in  2005.  This  environment  was  used  to  look  at  how 
intelligence  reports  of  varying  reliability  influenced 
confidence  in  those  reports.  Thirty  participants  replicated 
Tactical  Operation  Centers  (TOCs)  in  a  division  and  2 
brigades  tasked  with  maintaining  security  in  an  urban 
setting  following  combat  operations.  Participants  received 
intelligence  briefings  prior  to  each  of  eight  2  1/2  hour 
trials.  Four  trials  presented  the  base  condition  with  1 
accurate  intelligence  report  and  2  bogus  reports.  The 
remaining  4  trials  presented  the  enhanced  condition  with  2 
accurate  intelligence  reports  and  2  bogus  reports. 
Participants  knew  that  2  reports  in  each  condition  were 
bogus.  No  significant  difference  was  found  between 
participant’s  confidence  in  intelligence  reports  prior  to 
each  trial  when  comparing  the  base  condition  and  the 
enhanced  condition  suggesting  that  a  higher  proportion  of 
bogus  reports  did  not  result  in  reduced  confidence.  Also, 
confidence  levels  did  not  decrease  over  trials. 
Interestingly,  most  participants  tended  to  treat  most 
intelligence  reports  as  if  they  were  good  intelligence 
despite  knowing  that  at  least  half  of  the  reports  were 
bogus. 

1.  INRODUCTION 

Military  analysts  know  that  information  in 
intelligence  reports  may  be  inaccurate  or  incomplete. 
Does  this  uncertainty  lead  to  distrust  in  intelligence 
reports?  If  so,  will  participants’  trust  in  intelligence 
information  decline  when  they  know  that  a  higher  percent 
of  the  Intel  is  bogus? 

Breznitz  (1984)  examined  how  spurious 
information  influenced  trust  in  subsequent  information. 
He  called  this  phenomenon  where  operators  learned  to 
distrust  systems  that  had  been  unreliable  in  the  past  the 
cry-wolf  phenomenon.  Breznitz  proposed  the  Theory  of 
Initial  Credibility  that  states  that  loss  of  credibility 
depends  on  the  initial  level  of  credibility  before  receiving 
any  false  information.  Additionally,  his  Law  of 
Interwarning  Similarity  states  that  the  more  similar  true 
and  false  signals,  the  greater  the  credibility  loss  for  true 
signals.  Based  on  these  theories,  the  best  way  to  prevent 


the  cry-wolf  phenomenon  would  be  to  have  a  system  that 
was  believed  to  be  highly  credible  and  have  participants 
who  are  readily  able  to  discriminate  between  true  and 
false  signals. 

This  research  looked  at  how  analysts  rated  the 
reliability  of  intelligence  reports  when  they  knew  that  !4 
or  2/3 ’s  of  the  reports  received  were  bogus.  Since  all 
intelligence  reports  were  based  on  actual  reports,  it  was 
difficult  for  participants  to  determine  the  veracity  of 
reports.  According  to  Breznitz’  theories,  operators  should 
not  have  a  high  level  of  confidence  in  the  reports  and 
would  have  difficulty  determining  which  were  true.  This 
would  result  in  increased  distrust  in  subsequent  reports. 

2.  METHOD 

2.1  Participants 

Active  duty  military  and  retired  military 
personnel  from  the  United  States  Army,  Marines,  Navy, 
and  Air  Force  participated  in  this  human-in-the-loop 
experiment.  Although  all  had  military  experience,  few 
worked  in  the  intelligence  field.  Participants  assumed  the 
positions  of  decision  makers  in  tactical  operations  centers 
(TOCs)  by  interacting  with  the  JSAF  simulation  and  with 
each  other.  Situational  awareness  was  maintained  by 
placing  markers  on  electronic  maps  and  communication 
to  other  TOCs  via  text  chat  over  the  Info-work  Space 
(IWS).  Information  from  sensors  was  obtained  through 
stationary  sensors  and  aerial  sensors  that  could  be 
directed  to  areas  of  interest. 

2.2  Physical  Set-up 

Participants  were  located  in  3  separate  rooms  composed 
of  the  following  positions: 

Division  Command  Information  Center  (CIC): 
Commander,  Operations,  Air,  Intelligence  (2),  Collection 
Manager  (2),  Counter  Intelligence/Human  Intelligence 
(CI/HUMINT),  (2),  Sensors  (2), 

Brigade  Base  Defense  Operations  Center:  Commander, 
Operations,  Intelligence,  CI/HUMINT  Sensors  (2) 

Brigade  TOC:  Commander,  Operations,  Asst  Operations, 
Intelligence  (2),  CI/HUMINT. 


2.3  Test  conditions 

UR  2005  was  a  laboratory  experiment  consisting 
of  eight  2  1/2  hour  trials  presented  in  a  simulated 
environment  with  participants  asked  to  make  decisions 
based  on  information  from  simulated  sensors  and 
intelligence  briefings  and  injects.  Trials  contained  a  mix 
of  attack  types  (IED,  mortar,  ambush)  against  targets 
such  as  convoys,  local  police,  and  coalition  forces.  The 
Red  team  executed  152  missions  in  16  trials  conducted 
over  two  weeks.  Nine  or  ten  missions  were  conducted  per 
trial  with  each  mission  consisting  of  3-to-4  activities. 

Two  conditions  of  intelligence  were  presented. 
Prior  to  each  of  eight  2  1/2  hour  trials,  participants 
received  an  intelligence  brief  containing  either  3  or  4 
intelligence  reports.  In  the  base  case,  three  reports  were 
given,  with  2  of  the  3  being  bogus.  In  the  enhanced 
condition,  4  reports  were  provided  with  2  being  accurate 
intelligence  and  2  bogus.  Participants  knew  that  2  reports 
in  each  condition  were  bogus.  Additional  intelligence 
updates  were  provided  on  both  the  accurate  and  bogus 
intelligence  during  the  trial.  An  explosion  event  (mortar 
fire,  IED,  ambush)  occurred  for  each  accurate 
intelligence  report  which  greatly  increased  the  chance 
that  participants  would  detect  activity. 

2.4  Data  collection 

After  receiving  the  intelligence  briefing  and 
before  beginning  the  trial,  participants  were  asked  to 
evaluate  the  3  or  4  intelligence  reports  and  rate  their 
confidence  that  each  report  was  true  and  rate  the 
importance  of  the  activity  reported.  At  the  conclusion  of 
each  trial,  participants  reported  whether  each  report  was 
true,  what  lead  them  to  believe  that  it  was  true  or  false, 
and  how  successful  they  were  in  responding  to  the  threat 
reported.  Participants  then  were  debriefed  on  the 
accuracy  of  the  intelligence  and  how  well  they  succeeded 
in  using  the  intelligence. 

To  summarize,  participants  knew  before  each 
trial  that  2  intelligence  reports  were  bogus,  and  they  were 
informed  which  were  bogus  at  the  end  of  each  trial.  It 
was  hypothesized  that  participant’s  confidence  in  the 
intelligence  reports  would  decline  over  the  8  trials  and 
that  their  confidence  in  the  base  condition,  where  2/3  of 
the  intelligence  reports  were  bogus,  would  be  lower  than 
in  the  enhanced  condition  where  1/2  of  the  reports  were 
bogus. 

3.  RESULTS 


fifth  trials  (see  Figure  1).  Both  of  these  trials  were  the 
enhanced  condition. 


Confidence  in  Intelligence 


Trial  Number 


Figure  1.  Confidence  rating  in  intelligence  reports  prior 
to  trial.  Trial  1  is  significantly  higher  than  Trial  5.  Grey 
bars  indicate  enhanced  condition.  Black  bars  indicate 
base  condition. 

Interestingly,  participants  tended  to  respond  that 
they  were  very  confident  or  somewhat  confident  in  the 
intelligence  received  (58  percent  of  the  time)  and  rarely 
rated  their  confidence  level  as  not  very  or  not  at  all 
confident  (14  percent  of  the  time).  In  other  words,  they 
tend  to  treat  most  intelligence  as  if  it  were  good 
intelligence  despite  knowing  that  at  least  half  of  the 
reports  were  bogus  (see  Table  1).  Participants  did  not 
discriminate  between  true  and  bogus  reports  based  solely 
on  the  intelligence  report  itself.  They  identified  29 
percent  of  the  true  reports  as  probably  true  and  29  percent 
of  the  bogus  reports  as  probably  true. 

Table  1.  Participant’s  report  of  their  confidence  that 
intelligence  reports  were  true  at  the  beginning  of  each 
trial.  Twenty-eight  percent  of  the  time  they  were 
undecided. 


True  (Truth) 

Bogus  (Truth) 

True  (Reported) 

29  percent 

29  percent 

Bogus  (Reported) 

4  percent 

10  percent 

At  the  end  of  each  trial,  participants  were  asked 
if  the  intelligence  reports  were  true  (see  Table  2).  Again, 
there  was  a  tendency  to  report  that  the  intelligence  reports 
were  true  and  the  activity  did  take  place,  although  their 
level  of  uncertainty  increased. 

Table  2.  Participants  report  of  whether  the  intelligence 
reports  were  true  following  each  trial.  Forty-one  percent 
reported  that  they  did  not  know  if  the  activity  occurred. 


Results  did  not  support  our  hypothesis. 

True  (Truth) 

Bogus  (Truth) 

Confidence  in  intelligence  reports  did  not  decline  over 

True  (Reported) 

28  percent 

20  percent 

time.  A  significant  difference  in  confidence  in 

Bogus  (Reported) 

2  percent 

8  percent 

intelligence  reports  was  found  only  between  the  first  and 


Based  on  observer  report  and  IWS 
communication,  all  attack  missions,  whether  intelligence 
was  provided  or  not,  were  detected.  Since  all  missions 
associated  with  an  intelligence  report  involved  an  attack 
activity,  they  all  were  detected.  Table  2  indicates  that 
participants  were  not  always  aware  the  mission  was 
detected.  Additionally,  they  reported  that  20  percent  of 
the  bogus  reports  were  true. 

4.  SUMMARY 

Contrary  to  the  prediction  that  high  levels  of 
bogus  intelligence  would  lead  to  decreased  confidence  in 
intelligence  over  the  trials,  no  decline  in  confidence  was 
seen.  Each  intelligence  report  described  a  potential  threat 
and  each  threat  was  responded  to  as  if  it  were  real. 


These  findings  highlight  the  difficulty  in 
interpreting  actions  in  an  environment  fraught  with 
uncertainty.  If  participants  responded  to  an  intelligence 
report  and  nothing  happened  was  it  because  they  thwarted 
the  enemy’s  activities  or  was  the  intelligence  incorrect. 
Blatt  (2005)  noted  that  “Wrong  information  may  be  more 
lethal  than  missing  information  because  it  might  not  be 
recognized  that  the  information  is  wrong.”  (p.  65). 
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